Touch screen game controller

ABSTRACT

Aspects of the present invention allow a touch screen device to operate as a gamepad for a video game running on a separate gaming device. Aspects of the present invention provide a virtual control interface through the touch screen that includes virtual game controls that perform the same function as a directional pad, buttons, triggers, and sticks found on a gamepad or other game controller. In one aspect, the virtual control interface provides an output that is substantially similar to an output generated by a gamepad able to interact with the gaming device. Substantially similar outputs allow a gaming device to respond to a functional control from a gamepad and an equivalent functional control from the virtual control interface the same way and without translation. Aspects of the present invention can provide different virtual control interfaces for different video game titles, different genres, and based on user customization.

RELATED APPLICATIONS

This application is a divisional of U.S. patent application Ser. No.14/145,922, filed Dec. 31, 2013, entitled “TOUCH SCREEN GAMECONTROLLER,” the entirety of which is hereby incorporated by reference.

BACKGROUND

Video games are often controlled using a sophisticated game controllerthat includes multiple joy sticks and 10-20 separate buttons. The gamecontrollers may vibrate and emit sounds in response to signals receivedfrom the game console. Some game controllers include a built-in displaythat provides supplemental game content.

SUMMARY

This summary is provided to introduce a selection of concepts in asimplified form that are further described below in the detaileddescription. This summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used in isolation as an aid in determining the scope of the claimedsubject matter.

Aspects of the present invention allow a touch screen device to operateas a gamepad for a video game played on that same device (e.g. a phoneor a tablet) or on a separate gaming device (e.g., a game console, aPC). Aspects of the present invention provide a virtual controlinterface that includes virtual game controls that perform the samefunction as a directional pad, buttons, triggers, and sticks found on agamepad. The virtual game controls can be ergonomically located on thetouch screen to accommodate different hand sizes and methods of holdinga touch screen device. The control's location may be dynamicallyadjusted accommodate individual users hand size, preferred handlocation, preferred interaction style, and grip style. Location can beadjusted based on how the user is holding the touch screen device.

In one aspect, the virtual control interface provides an output that issubstantially similar to an output generated by a gamepad able tointeract with the gaming device. Substantially similar outputs allow agame and a gaming device to respond to a functional control from agamepad and an equivalent functional control from the virtual controlinterface in the same way and without translation. For example, pushingthe green reference button on the gamepad and pushing the correspondingvirtual green reference button on the virtual control interface willresult in the gaming device and the game receiving the same functionalsignal. The signal generated by the touch screen device and gamepad maydiffer in some ways, for example, different devices may generatedifferent identification information.

Aspects of the present invention can provide different virtual controlinterfaces for different video game titles. In one aspect, the user canselect a virtual control interface from a group of appropriate virtualcontrol interfaces. Virtual control interfaces may be optimized fordifferent characteristics of a game. For example, virtual controlinterfaces may be optimized for a game genre. In one aspect, the virtualcontrol interface may be selected for a particular game title. In oneaspect, the virtual control interface is user customizable. The user maymove virtual game controls around on the interface and resize them asdesired. The customized virtual game control interface may be sharedwith friends and/or saved for later retrieval. The user can createcustomized virtual shortcuts that represent a combination ofinteractions with a gamepad. For example, a single virtual game controlcould cause an action in the game that requires multiple gamepadcontrols to be pressed.

To replicate a gamepad's tactile button feedback, a visual indication oraudible indication may be provided to let the user know that hesuccessfully interacted with a virtual game control. For example, theperimeter of a touch screen may glow faintly blue in response toselection of a blue button. Arrows or other visual indicia can providefeedback indicating selection of a virtual direction pad or virtualstick.

BRIEF DESCRIPTION OF THE DRAWINGS

Aspects of the invention are described in detail below with reference tothe attached drawing figures, wherein:

FIG. 1 is a block diagram of an exemplary computing environment suitablefor implementing aspects of the invention;

FIG. 2 is a diagram of a gaming environment, in accordance with anaspect of the present invention;

FIG. 3 is a diagram of a remote computing environment for gaming, inaccordance with an aspect of the present invention;

FIG. 4 is a diagram of a standard gamepad;

FIG. 5 is a diagram of an exemplary hand position suitable to manipulatea virtual control interface on a touch screen device, in accordance withan aspect of the present invention;

FIG. 6 is a diagram of an exemplary hand position suitable to manipulatea virtual control interface on a touch screen device, in accordance withan aspect of the present invention;

FIG. 7 is a diagram of a gamepad oriented virtual control interface withannotations, in accordance with an aspect of the present invention;

FIG. 8 is a diagram of a first-person shooter oriented virtual controlinterface with annotations, in accordance with an aspect of the presentinvention;

FIG. 9 is a diagram of a racing oriented virtual control interface withannotations, in accordance with an aspect of the present invention;

FIG. 10 is a diagram of a fighting oriented virtual control interfacewith annotations, in accordance with an aspect of the present invention;

FIG. 11 is a diagram of a single thumb interface showing movementcontrol, in accordance with an aspect of the present invention;

FIG. 12 is a diagram of a single thumb interface showing a button push,in accordance with an aspect of the present invention;

FIGS. 13-14 are diagrams showing control relocation, in accordance withan aspect of the present invention;

FIG. 15 is a diagram of a WASD interface, in accordance with an aspectof the present invention;

FIG. 16 is a flow chart showing a method for using a touch screencomputing device to control a video game, in accordance with an aspectof the present invention;

FIG. 17 is a flow chart showing a method for using a touch screencomputing device to control a video game, in accordance with an aspectof the present invention; and

FIG. 18 is a flow chart showing a method of providing a virtual controlinterface, in accordance with an aspect of the present invention.

DETAILED DESCRIPTION

The subject matter of aspects of the invention is described withspecificity herein to meet statutory requirements. However, thedescription itself is not intended to limit the scope of this patent.Rather, the inventors have contemplated that the claimed subject mattermight also be embodied in other ways, to include different steps orcombinations of steps similar to the ones described in this document, inconjunction with other present or future technologies. Moreover,although the terms “step” and/or “block” may be used herein to connotedifferent elements of methods employed, the terms should not beinterpreted as implying any particular order among or between varioussteps herein disclosed unless and except when the order of individualsteps is explicitly described.

Aspects of the present invention allow a touch screen device to operateas a gamepad for a video game played on that same device (e.g., a phoneor a tablet) or on a separate gaming device (e.g., a game console, aPC). Exemplary gaming devices include game consoles and personalcomputers. Exemplary gamepads include the Xbox 360 controller, Xbox Onecontroller, Wii Remote, Wii U™ GamePad and Sony's DualShock®3.Generally, a gamepad will include multiple input controls, including adirectional pad, buttons, triggers, and one or more sticks. A game padcan include between 10 and 20 inputs.

Aspects of the present invention provide a virtual control interfacethat includes virtual game controls that perform the same function as adirectional pad, buttons, triggers, and sticks found on a gamepad. Thevirtual game controls can be ergonomically located on the touch screento accommodate different hand sizes and methods of holding a touchscreen device. The control's location may be dynamically adjustedaccommodate individual users hand size, preferred hand location,preferred interaction style, and grip style. Location can be adjustedbased on how the user is holding the touch screen device.

In one aspect, the virtual control interface provides an output that issubstantially similar to an output generated by a gamepad able tointeract with the gaming device. Substantially similar outputs allow agaming device to respond to a functional control from a gamepad and anequivalent functional control from the virtual control interface in thesame way and without translation. For example, pushing the greenreference button on the gamepad and pushing the corresponding virtualgreen reference button on the virtual control interface will result inthe gaming device and the game receiving the same functional signal.

The signal generated by the virtual control interface and that of a realgamepad may differ in some ways, for example, different devices maygenerate different identification information. In another aspect, thetouch screen device can communicate commands through a differentcommunication channel, such as WI-FI, from the one used by the gamepad,such as infrared. When different channels are used, the functionalinstruction generated by the touch screen device may still be similar tothe functional instruction generated by a gamepad.

Aspects of the present invention can provide different virtual controlinterfaces for different video game titles. In one aspect, the user canselect a virtual control interface from a group of appropriate virtualcontrol interfaces adapted for a particular touch screen device. Thevirtual control interfaces may be purchased and downloaded to a touchscreen device for use. Players may be given an opportunity to rateavailable virtual control interfaces. The ratings may be shared withfriends for display in conjunction with a description of a virtualcontrol interface at a point where the virtual control interface ispurchased and/or downloaded.

Virtual control interfaces may be optimized for differentcharacteristics of a game. For example, virtual control interfaces maybe optimized for a game genre. A first virtual control interface may beoptimized for racing games, a second virtual control interface may beoptimized for shooting games, and a third virtual control interface maybe optimized for fighting games. Other virtual control interfaces may bemore general. In one aspect, a general control interface is optimizedduring play by analyzing which virtual controls the user is interactingwith. The virtual controls may be resized and relocated to make the mostcommonly used virtual controls more conveniently located and sized.

In one aspect, the virtual control interface may be selected for aparticular game title. Controls found on a game that are not used by thegame title may be left off or were omitted from the virtual controlinterface. Similarly, the most frequently used controls within the gametitle may be conveniently located and sized. In addition to controlsfound on a gamepad, special controls may be added to a virtual controlinterface. The special controls may represent a combination of buttonpushes on a gamepad. In other words, selecting a virtual control mayresult in a functional instruction that corresponds to a functionalinstruction that would result from interacting with multiple functionson a gamepad. In addition controls may be given names that aremeaningful within that particular title and help the user more easilylearn how to play the game. Controls can also be styled in a way that isaligned with the look and feel of the game.

In one aspect, the virtual control interface is user customizable. Theuser may move virtual game controls around on the interface and resizethem as desired. The customized virtual game control interface may beshared with friends and/or saved for later retrieval. The user cancreate customized virtual shortcuts that represent a combination ofinteractions with a gamepad. For example, a single virtual game controlcould combine movement with automatic weapon firing.

As mentioned, the virtual control interface may also be customized for auser in an automated fashion by analyzing the user's interactions withthe virtual interface. Virtual controls may be resized and relocated toaccommodate the user. In addition, various virtual controls are removedand resized during game play based on user interaction. For example, agame control can be located in response to the user's first touch on thescreen. Each time users removes their hand from the touch screen, thevirtual control is relocated upon the next touch. This arrangement maybe suitable for virtual controls that replace a stick or direction pad.

To replicate a gamepad's tactile button feedback, a visual indication oraudible indication may be provided to let the user know that hesuccessfully interacted with a virtual game control. For example, theperimeter of a touch screen may glow faintly blue in response toselection of a blue button. Arrows or other visual indicia can providefeedback indicating selection of a virtual direction pad or virtualstick.

Aspects of the present invention may translate gamepad feedbackinstructions from the gaming device into an output that is adapted forthe touch screen device. For example, a feedback instruction to “rumble”may be translated into a screen flash or other visual output or audiooutput. Alternatively, the touch screen device may vibrate if avibration function is included within the touch screen device. Audiofeedback may be output through the touch screen device's speakers.

In one aspect, developers can provide customized virtual game controlsfor their games. Developers may improve the effectiveness of acontroller design by providing configuration settings for the controllerand/or providing run-time hints. For example, a developer can configurebutton placement, size, types of buttons, what buttons are needed,combos, and the like. At run-time, a soccer game could give the systeman annotation indicating whether the player is on offense vs. defense,which could change the function and look (e.g., names) of the buttons.Developers can also provide art that is complimentary to the look andfeel of the game.

Having briefly described an overview of aspects of the invention, anexemplary operating environment suitable for use in implementing aspectsof the invention is described below.

Exemplary Operating Environment

Referring to the drawings in general, and initially to FIG. 1 inparticular, an exemplary operating environment for implementing aspectsof the invention is shown and designated generally as computing device100. Computing device 100 is but one example of a suitable computingenvironment and is not intended to suggest any limitation as to thescope of use or functionality of the invention. Neither should thecomputing device 100 be interpreted as having any dependency orrequirement relating to any one or combination of componentsillustrated.

The invention may be described in the general context of computer codeor machine-usable instructions, including computer-executableinstructions such as program components, being executed by a computer orother machine, such as a personal data assistant or other handhelddevice. Generally, program components, including routines, programs,objects, components, data structures, and the like, refer to code thatperforms particular tasks or implements particular abstract data types.Aspects of the invention may be practiced in a variety of systemconfigurations, including handheld devices, consumer electronics,general-purpose computers, specialty computing devices, etc. Aspects ofthe invention may also be practiced in distributed computingenvironments where tasks are performed by remote-processing devices thatare linked through a communications network.

With continued reference to FIG. 1, computing device 100 includes a bus110 that directly or indirectly couples the following devices: memory112, one or more processors 114, one or more presentation components116, input/output (I/O) ports 118, I/O components 120, and anillustrative power supply 122. Bus 110 represents what may be one ormore busses (such as an address bus, data bus, or combination thereof).Although the various blocks of FIG. 1 are shown with lines for the sakeof clarity, in reality, delineating various components is not so clear,and metaphorically, the lines would more accurately be grey and fuzzy.For example, one may consider a presentation component such as a displaydevice to be an I/O component 120. Also, processors have memory. Theinventors hereof recognize that such is the nature of the art, andreiterate that the diagram of FIG. 1 is merely illustrative of anexemplary computing device that can be used in connection with one ormore aspects of the invention. Distinction is not made between suchcategories as “workstation,” “server,” “laptop,” “handheld device,”etc., as all are contemplated within the scope of FIG. 1 and refer to“computer” or “computing device.”

Computing device 100 typically includes a variety of computer-readablemedia. Computer-readable media can be any available media that can beaccessed by computing device 100 and includes both volatile andnonvolatile media, removable and non-removable media. By way of example,and not limitation, computer-readable media may comprise computerstorage media and communication media. Computer storage media includesboth volatile and nonvolatile, removable and non-removable mediaimplemented in any method or technology for storage of information suchas computer-readable instructions, data structures, program modules orother data.

Computer storage media includes RAM, ROM, EEPROM, flash memory or othermemory technology, CD-ROM, digital versatile disks (DVD) or otheroptical disk storage, magnetic cassettes, magnetic tape, magnetic diskstorage or other magnetic storage devices. Computer storage media doesnot comprise a propagated data signal.

Communication media typically embodies computer-readable instructions,data structures, program modules or other data in a modulated datasignal such as a carrier wave or other transport mechanism and includesany information delivery media. The term “modulated data signal” means asignal that has one or more of its characteristics set or changed insuch a manner as to encode information in the signal. By way of example,and not limitation, communication media includes wired media such as awired network or direct-wired connection, and wireless media such asacoustic, RF, infrared and other wireless media. Combinations of any ofthe above should also be included within the scope of computer-readablemedia.

Memory 112 includes computer-storage media in the form of volatileand/or nonvolatile memory. The memory 112 may be removable,nonremovable, or a combination thereof. Exemplary memory includessolid-state memory, hard drives, optical-disc drives, etc. Computingdevice 100 includes one or more processors 114 that read data fromvarious entities such as bus 110, memory 112 or I/O components 120.Presentation component(s) 116 present data indications to a user orother device. Exemplary presentation components 116 include a displaydevice, speaker, printing component, vibrating component, etc. I/O ports118 allow computing device 100 to be logically coupled to other devicesincluding I/O components 120, some of which may be built in.Illustrative I/O components 120 include a microphone, joystick, gamepad, satellite dish, scanner, printer, wireless device, etc.

Exemplary Online Gaming Environment

Turning now to FIG. 2, an online gaming environment 200 is shown, inaccordance with an aspect of the present invention. The online gamingenvironment 200 comprises various game clients connected through anetwork 220 to a game service 230. Exemplary game clients include a gameconsole 210, a touch screen device 212, and a personal computer 214. Useof other game clients, such as smart phones, are also possible. The gameconsole 210 may have one or more game controllers communicativelycoupled to it. In one aspect, the touch screen device 212 may act as aninput device for a game console 210 or a personal computer 214 using avirtual control interface. Network 220 may be a wide area network, suchas the Internet.

The controllers associated with game console 210 include game pad 231,tablet 232, headset 236, and depth camera 234. A game console may beassociated with control devices that generate both a rich input and abasic input. Individual controllers are capable of generating differentkinds of inputs and a single controller could generate both a rich inputand an basic input.

The game pad 231 may be capable of generating basic control signals,such as those generated by button selections and joystick movement.Movement data such as that generated by accelerometers and gyros withinthe game pad 231 may be examples of rich sensory data. In someimplementations, the movement data is not considered a rich sensorydata.

Tablet 232 is shown coupled directly to the game console 210, but theconnection could be indirect through the Internet or a subnet. In oneaspect, the game service 230 helps make a connection between the tablet232 and touch screen device 212 and the game console. The tablet 232 iscapable of generating numerous input streams and may also serve as adisplay output mechanism. In addition to being a primary display, thetablet 232 could provide supplemental game information related toinformation shown on a primary display coupled to the game console 210,or simply be a control surface. The input streams generated by thetablet 232 include video and picture data, audio data, movement data,touch screen data, and keyboard input data.

The headset 236, captures audio input from a player and the player'ssurroundings and may also act as an output device if it is coupled witha headphone or other speaker.

The depth camera 234 generates a depth cloud used as a control input.The depth camera 234 may an use infrared camera to determine a depth, ordistance from the camera for each pixel captured. Stereoscopic depthcameras are also possible. In addition, the depth camera 234 may capturea typical color stream or picture. The depth camera 234 may have severalimage gathering components. For example, the depth camera 234 may havemultiple cameras.

Various combinations of components shown in FIG. 2 can generate avirtual control interface and/or execute code needed to play avideogame. For example, game code could be executed by game console 210and controlled by a virtual control interface generated by tablet 212 ortablet 232. In one aspect, game code could be executed by PC 214 andcontrolled by a virtual control interface generated by tablet 212 ortablet 232. In another aspect, game code to be executed by game service230 producing a rendered video game image that is displayed on PC 214, atelevision (not shown), on tablet 212 or tablet 232. The video gameimage can be simultaneously displayed on tablet 212 or tablet 232 alongwith a virtual control interface. In another aspect, tablet 212 ortablet 230 to execute the game code and generates a virtual controlinterface that controls the game. In one aspect, the game code andvirtual control interface is generated by tablet 212 and a video gameimage is communicated to a separate device, such as PC 214 or atelevision (not shown). In another alternative, the videogame code isexecuted on tablet 212 and displayed through tablet 212 along with avirtual control interface. Other combinations can be possible includingsplit render and execution scenarios where part of the videogame code isexecuted by different devices and combined into a single video gameimage.

Game service 230 may comprise multiple computing devices communicativelycoupled to each other. In one aspect, the game service 230 isimplemented using one or more data centers. The data centers may bespread out across various geographic regions including cities throughoutthe world. In this scenario, the game clients may connect to the closestdata centers. Aspects of the present invention are not limited to thissetup.

The game service 230 allows the game to be executed within the computingdevices provided by the game service 230. A communication sessionbetween the game service and game clients carries input traffic to thegame service 230 and can return a rendered game image and/or other gameoutput.

The virtual control interface may run on the touch screen device 212 topresent and interface and communicate output to the gaming device (e.g.game console 210, PC 214). An application can also run on the gameconsole 210 or PC 214 to receive the output from the touch screen deviceand translate into commands understood by a game title.

Exemplary Gaming Device and Game Service for Virtual Interface Control

Turning now to FIG. 3, an exemplary remote gaming environment 300 isshown, in accordance with an aspect of the present invention. The remotegaming environment 300 includes a gaming device 310 communicativelycoupled to a game server 340 through a network 330. In one aspect, thenetwork may be the Internet. The gaming device 310 is connected to agamepad 312, a touch screen device 314, and a display 316. Other gameinput devices may be used with the gaming device 310, includingkeyboards, a mouse, a touch pad, a touch screen, a microphone forreceiving voice commands, a depth camera, a video camera, and atrackball. Aspects of the present invention are not limited to theseinput devices. The display 316 is capable of displaying video gamecontent. For example, the display 316 may be a television or computerscreen.

The touch screen device 314 runs a virtual control interface program.The virtual control interface program allows the touch screen device 314to output a suitable virtual control interface. The virtual controlinterface program may translate inputs received through the touch screeninto functional commands understood by a gaming device, and/or by avirtual interface component 321 on the gaming device. The virtualcontrol interface may control a game running on the touch screen device314, the gaming device 310, or the game service 340. Controls can betranslated into an appropriate format as needed for consumption by thedevice running the game. The functions of the virtual control interfaceprogram are described in detail with reference to FIGS. 16, 17, and 18.Exemplary virtual control interfaces and virtual controls will beillustrated subsequently with reference to FIGS. 5-15.

The gaming device 310 is a computing device that is able to executevideo games. The gaming device 310 could be laptop computer or a gameconsole. The gaming device 310 includes an operating environment 320, avirtual interface component 321, a game execution environment 322, avirtual interface library, and a game data store 324. Other componentsof the gaming device 310 are not shown for the sake of simplicity.

The operating environment 320 may be provided by an operating systemthat manages the hardware and provides services to applications runningon the gaming device 310. The operating environment may allocate clientresources to different applications as part of game and communicationfunctions.

The game data store 324 stores downloaded games, game samples, and/orpartially downloaded games. Games may be downloaded in playable blocks.To play a game, the game may need to be loaded from the game data store324 into active memory associated with the game execution environment322. The game data store 324 may also store player progress files.

The game execution environment 322 comprises the gaming resources on thedevice 310 required to execute instances of a game. The game executionenvironment 322 comprises active memory along with computing and videoprocessing resources. The game execution environment 322 receives gamingcontrols and causes the game to be manipulated and progressed accordingto game programming In one aspect, the game execution environment 322outputs a rendered video stream that is communicated to the display 316.

The virtual interface component 321 can enable a video game titlerunning within game execution environment 322 or game executionenvironment 348 to be controlled by a virtual control interface runningon touch screen device 314. The virtual interface component 321 mayfacilitate communications between the touch screen device and the gamingdevice 310. Communications between the gaming device 310 touch screendevice 314 can be bidirectional. The virtual interface component 321 maytranslate functional instructions received from the touch screen deviceinto commands that can be fed into an understood by the game executionenvironment 322 to manipulate the videogame. Likewise, the virtualinterface component 321 may receive feedback from the game executionenvironment 322 and translated into a feedback instruction that isconsumable by the touch screen device. For example, a rumble commandcould be translated into a command to activate the vibration function onthe touch screen device.

The virtual interface component 321 can also facilitate customization ofa virtual control interface and storage of the customized virtualcontrol interface within virtual interface library 323 or virtualinterface library 350. The virtual interface component 321 may alsofacilitate the storage of user preferences.

The game server 340 comprises a connection manager 342, a player profiledata store 344, a game availability manager 346, a game executionenvironment 348, a virtual interface library 350, and a virtualinterface distribution component 352. Though depicted as a single box,the game server 340 could be a data center that comprises numerousmachines, or even several data centers. Several of the servers could actas clients for a central server that coordinates the game experience.

The connection manager 342 builds a connection between the gaming device310 and the service 340. The connection manager 342 may also providevarious authentication mechanisms to make sure that the user isauthorized to access the game service provided by the service 340. Theconnection manager may provide security, encryption, and authenticationinformation to servers and virtual machines as they are added to a gamesession.

The player profile data store 344 may work in conjunction with theconnection manager 342 to build and store player information. Part ofthe player profile may comprise demographic and financial informationsuch as a player's name, address and credit card information or othermechanism for paying for or purchasing games and experiences provided bythe game service.

In addition, the player profile data store 344 may store a player'sprogress within an individual game. A player's score, achievements, andprogress through game levels may be stored. Further, the player profiledata store 344 may store information about individual player preferencessuch as language preferences. The player may access the game levelinformation from multiple clients. For example, the player's progresscould be accessed from a friend's game console or on the player's mobiledevice.

The player profile data store 344 may also store a usage history for theindividual player. A player's history of purchasing games, virtualcontrol interfaces, sampling games, or playing games through a gameservice that does not require the purchase of the games may be stored. Arecord of a player's favorite virtual control interfaces for variousgames may be tracked.

The game availability manager 346 analyzes the usage data to determine,among other things, how many standby instances of a particular gametitle should be available. In general, games with a high demand willhave more standby instances of a game available. Loading a game intoactive memory to create a standby instance may take a minute or two;thus, games with high churn in and out may also require more standbyinstances of games to be available. The time it takes to create astandby instance of a particular game title also should be considered.Games that load comparatively quickly may require less standby instancesbecause additional game instances can be generated more quickly asdemand changes. In other words, games with a slower load time mayrequire more available standby games.

The game execution environment 348 comprises the gaming resourcesrequired to execute instances of a game. The game execution environment348 comprises active memory along with computing and video processing.The game execution environment 348 receives gaming controls, such asreduced controller input, through an I/O channel and causes the game tobe manipulated and progressed according to its programming. In oneaspect, the game execution environment 348 outputs a rendered videostream that is communicated to the game client. In other aspects, thegame execution environment 348 outputs game geometry, or otherrepresentations, which may be combined with local objects on the gamingclient to render the gaming video.

The virtual interface library 350 stores available virtual controlinterfaces. Virtual control interfaces may be created by developers andassociated with game titles. The virtual control interfaces may bepurchased from the virtual interface library through the virtualinterface distribution component 352. Virtual control interfaces mayalso be created by users and stored for subsequent use. User-createdvirtual control interfaces may also be shared with friends or anyoneelse depending on the creators distribution preferences. Some virtualcontrol interface may be given away for free or for promotionalconsideration.

The library 350 can store virtual control interfaces using aclassification system and a ranking system. The classification systemcan be used to classify an interface by game genre, game title,developer, hardware, or other characteristic. The ranking system can bebased on user feedback. The feedback could be explicit or implicitlyderived from usage. A frequently downloaded interface can be ranked morehighly than in frequently downloaded interface.

The virtual interface distribution component 352 manages a cloud-basedecosystem of virtual control interfaces. The distribution component 352can promote virtual control interfaces to users through recommendations,advertisements, rankings, and user comments or reviews. In one aspect,the distribution component 352 may analyze a user's gaming history andrecommend virtual control interfaces that may be of use.

The virtual interface distribution component 352 can also facilitatecommunication of customized virtual control interfaces between friends.A first user can upload a virtual control interface to the virtualinterface library 350 and give permission for friends to access theinterface. In one aspect, friends or designated via a social network. Inother words, a user could specify that any relation within the usersocial network can have access to a virtual control interface shecreated. In one aspect, the user's social network comprises contactswithin an online gaming service. In another aspect, the user's socialnetwork comprises contacts within one or more dedicated social networks,such as Facebook.

The virtual interface distribution component 352 allows users to searchfor virtual control interfaces that may be of interest. The virtualcontrol interfaces may be ranked by popularity. The popularity of avirtual control interface may be ascertained by its frequency ofdownload, customer ratings, reputation of its creator, or through someother method. For example, a game developer may be assigned a betterreputation than an amateur. The search function can allow users tosearch for virtual control interfaces tailored to a particular gamegenre or even a particular game title.

The virtual interface distribution component 352 may provide a platformthat is downloadable to touch screen devices and gaming devices. Thedownloadable platform can be adapted for different operating systems andservices. In one aspect, the virtual control interfaces are applicationsthat run within the platform. In this way, different platforms may becreated for different operating systems or devices, but a single versionof the virtual control interface application could run across multipleplatforms, screen sizes, etc. This would allow a single version of thevirtual control interface to be operable across different types ofdevices and operating systems.

Turning now to FIG. 4, a gamepad controls on standard gamepad 400 isshown. Any control on a gamepad may be described as a gamepad controlherein. The gamepad 400 comprises several gamepad controls, including aleft stick 402 and a right stick 404. Gamepad controls on the top side401 of gamepad 400 includes a direction pad 406, a guide button 408, aback button 410, a start button 412, and face buttons 414, 416, 418, and420. Other gamepad controls on the gamepad 400 are a right bumper 422and a right trigger 424. The gamepad 400 also includes a left bumper 426and a left trigger 428. A

The gamepad 400 is ergonomically designed to be held by both the user'sright and left hand. The left hand may control the left stick 402 alongwith the left bumper 426 and the left trigger 428. A user may prefer tocontrol the left stick 402 with his left thumb, the left bumper 426 withhis left index finger, and the left trigger 428 with his left middlefinger. The right hand may control the right stick 404 along with theright bumper 422 and the right trigger 424. A user may prefer to controlthe right stick 404 with his right thumb, the right bumper 422 with hisright index, and the right trigger 424 with his right middle finger. Theface buttons 414, 416, 418, and 420 may also be controlled by the user'sright hand.

The different controls on the gamepad 400 control different features ofa game. For example, moving the left stick 402 may translate a playerthrough the game while the right stick 404 rotates the view. Gamedevelopers can establish each button's function in a game.

As used herein, the phrase “activating a gamepad control” meansinteracting with a gamepad control to cause a functional instruction tobe communicated from the gamepad 400. For example, pushing a referencebutton is an example of activating a gamepad control. In response topushing a reference button, the gamepad generates a functionalinstruction that is communicated to the gaming device. The gaming deviceinterprets the functional instruction as a push of the reference button.This instruction may be fed to a video game title and the gamemanipulated according to the consequence of pushing the referencebutton. Moving a stick is another example of activating a gamepadcontrol. Holding a stick or a button in a particular position or statemay cause multiple functional instructions to be generated by gamepad.

Turning now to FIG. 5, an exemplary hand position suitable to manipulatea virtual control interface on a touch screen device 500 is shown, inaccordance with an aspect of the present invention. A user holds thetouch screen device 500 with the bottom corner 502 of her right hand 510fit snugly in the crotch between her thumb 512 and forefinger. This canposition allows the user to interact with a virtual game controlslocated in the lower right-hand side of the touch screen device 500.

Turning now to FIG. 6, an exemplary hand position suitable for use witha virtual control interface on a touch screen device 600 is shown, inaccordance with an aspect of the present invention. The user holds theright side of touch screen device 600 in the crotch formed by his thumb602 and forefinger of his right hand 610. In this position, the user canslide 612 his thumb 602 up and down the touch screen. Though not shown,the user's fingers could contact a touch service located on the back oftouch screen device 600. For example, a keyboard could be folded behindthe touch screen device. Individual keys on the keyboard could be mappedto controls on a gamepad. Alternatively, areas of the touch keyboardcould be associated with a control such as a bumper or trigger.

Other hand positions are possible. For example, the user could lay atouch screen device on a surface and use multiple fingers to interactwith the virtual control interface.

FIGS. 7-15 provide examples of possible virtual control interfaces. Asmentioned, a touch screen device could execute game code and display avideo game image through the touch screen along with the virtual controlinterface that controls the game. In another aspect, the video gameimage generated by and/or displayed by a different device. For the sakeof simplicity, the video game image is not shown in the examples ofvirtual control interfaces provided herein.

Turning now to FIG. 7, a gamepad oriented virtual control interface 705with annotations is shown on a touch screen device 700, in accordancewith an aspect of the present invention. The virtual control interface705 includes many of the controls found on gamepad 400 describedpreviously. The virtual control interface 705 is divided into a rightzone 710 and a left zone 740. Generally, controls within the right zone710 are optimized for interaction with the user's right hand. Thecontrols within the left zone 740 are optimized for interaction with theuser's left hand. The user's hands are not shown, but the user could beholding the device or the device could be resting on a surface. Thestart button 738, the guide button 739, and the back button 742 arelocated at the top of the virtual control interface 705.

The right zone 710 includes the right stick group 711 and face buttons730, 732, 734, and 736. In one aspect, the face buttons are in a static.In other words, the face buttons can hold a location in the lower rightcorner of the virtual control interface 705 even as other controls mayhave a variable location. In one aspect, the arrangement of the facebuttons is inverted compared to the arrangement found on the gamepad.The face buttons are round, each the same size, and arranged in adiamond pattern. The face buttons may be displayed in a color found onan associated gamepad. The color of buttons on different game pads mayvary. The virtual controller interface can match the color of a gamepadselected by the user.

The right stick group 711 can start in a location where the userinitially interacts with the touch screen. For example, when the userplaces an implement, such as his thumb, middle, or index finger on thetouch screen at the right stick box 726. The virtual right stick control724 is then displayed within the right stick box 726. The virtual rightstick control 724 may be manipulated by moving a finger in differentdirections. Other functions within the right stick group 711 arearranged around the right stick box 726 and may be activated in avariety of ways.

The user may tap the virtual right trigger control 712 with an indexfinger while manipulating the virtual right stick control 724 with histhumb. Alternatively, the user may hold his finger, or other implement,on the virtual right trigger control 712 to generate a rapid firecommand that is the equivalent to pulling the trigger repeatedly inrapid succession. Alternatively, the user may control the virtual rightstick control 724 and “pull” the virtual right trigger control 712located within right trigger box 714 by dragging the virtual right stickcontrol 724 into the right trigger box 714. This allows the user tocontrol the virtual right stick control 724 and the virtual righttrigger control 712 with a single finger. While in the right trigger box714 the user may move her finger in a circular direction to manipulatethe virtual right stick control 724.

The user may tap the virtual right bumper control 720, located withinright bumper box 722. Holding a finger on the virtual right bumpercontrol 720 can produce a rapid push command that is the equivalent topushing the bumper repeatedly in rapid succession. Alternatively, theuser may control the virtual right stick control 724 and push thevirtual right bumper control 720 by dragging the virtual right stickcontrol 724 down into the right bumper box 722. This allows the user tocontrol the virtual right stick control 724 and the virtual right bumpercontrol 720 with a single finger. While in the right bumper box 722 theuser may move her finger in a circular direction to manipulate thevirtual rights to control 724.

The virtual right stick push control 716, located in right stick box718, causes the same command to be generated as depressing the rightstick on a gamepad. The virtual right stick push control 716 may betapped or held down by keeping a finger on the control. Alternatively,the user may control the virtual right stick control 724 and press thevirtual right stick push control 716, by dragging the virtual rightstick control 724 across into the right stick box 718. This allows theuser to control the virtual right stick control 724 and the virtualright stick push control 716 with a single finger. While in the rightstick box 718 the user may move her finger in a circular direction tomanipulate the virtual right stick control 724.

The left zone 740 includes a left stick group 743. The left stick group743 may have a floating location established the first time the usertouches the left zone 740. In one aspect, the virtual left stick control754 is centered on the first contact with the left zone 740. Prior tothe initial touch, an annotation can be presented in fighting the userto establish the location of the left stick group 743 by touching thescreen.

In one aspect, virtual control interface determines whether the initialtouch was by a thumb or finger. This detection can be made by analyzingthe shape and size of a contact zone with the touch screen interface.Games where the virtual left stick control 754 is optimally manipulatedby a thumb may reject initial touch not made by a thumb and insteadinstruct the user to hold the touch screen device in a way where histhumb may be used to manipulate the virtual left stick control 754. Asimilar analysis may be performed on other virtual controls to determinewhether the user is interacting with the controls in an optimal fashion.Areas for improvement may be communicated to a user to help the userefficiently use the virtual control interface 705.

The controls associated with the left stick group 743 may operate in thesame way as the controls within the right stick group 711. The variouscontrols may include different game aspects, but may be manipulated in asimilar manner The left stick group 743 includes the virtual left stickcontrol 754 located within left stick box 752, virtual left triggercontrol 744 located within left trigger box 746, virtual left stick push748 located within stick push box 750, and virtual left bumper control756 located within left bumper box 758.

The left zone also includes a virtual direction pad 760. The virtualdirection pad can be fixed to a specific location.

Aspects of the present invention, can provide annotations that help theuser understand how to use the virtual control interface. Annotation 770communicates that pushing the bumper and the left stick down will causethe user's character to perform a dropkick. The annotation could matchthe game title playing on a separate gaming device. Annotations can bespecific to particular challenges within a game. For example, anannotation, taking the form of a hint, may be provided upon observingthat the user is having difficulty overcoming a portion of a game. Theannotations could be provided upon the user achieving a new capability.Annotations can also be provided as an initial tutorial for the virtualcontrol interface 705.

Gamepad-specific annotations may also be provided. Gamepad-specificannotations let the user know how to generate inputs typically generatedby a gamepad. The gamepad-specific annotations may be applicable acrossgame titles and game genres. For example, annotation 772 instructs theuser to move the left stick control into the blue box (left trigger box746) to hold the trigger down.

Turning now to FIG. 8, a first-person shooter optimized virtual controlinterface 800 is shown, in accordance with an aspect of the presentinvention. Virtual control interface 800 includes many of the controlsfound on a gamepad. The controls are arranged and sized in a way that isoptimized for use with first-person shooter games, a popular genre. Asmentioned, genre specific virtual control interfaces may be generated bygame developers. Interfaces may also be generated by users thatcustomize a general interface to their liking.

Interface 800 includes a virtual right stick push control 830, a virtualright stick control 832, a virtual right stick bumper control 834, and avirtual right stick trigger control 836. Notice that the virtual rightstick trigger control 836 is larger than the virtual right stick bumpercontrol 834 because the trigger control is used more often in afirst-person shooter game than a bumper control. These four right stickcontrols form a right stick group that may be operated in a way similarto those described previously with reference to FIG. 7.

Virtual reference button A genre 840, virtual reference button X 842,virtual reference button B 844, and virtual reference button Y 846 arelocated in the lower right corner of virtual control interface 800. Inone aspect, these buttons are manipulated by interacting with the touchscreen at a point where these buttons are displayed. Maintaining acontinuous interaction with a button may cause the button to rapidfire.In another aspect, interacting with the button and then dragging acertain direction may cause the button to rapidfire at a rate that isproportional to the distance the button is dragged. The distance draggedmay stay within the button area or escape the area. The distance ismeasured from the first point of interaction within the button to asecond point where the movement stops. A visual or audible indicationmay be given upon dragging a distance associated with a maximum rate offire. For example, the perimeter of the screen may glow a colorassociated with the reference button when pushed and increase theillumination during the drag operation. Upon reaching the maximum rateof fire, the illuminated portion of the screen may flash to indicate themaximum has been reached.

The virtual control interface 800 also includes a left stick group andseries of buttons mimicking a direction pad. The left stick groupcomprises virtual bumper control 810, virtual left stick control 812,virtual left stick push 814, and virtual left trigger 816. Thesecontrols may be manipulated in a manner described previously withreference to FIG. 7. Virtual direction buttons are located below theleft stick group and replace a D-pad. The virtual direction buttonscomprise a virtual left control 818, a virtual up control 820, a virtualdown control 822, and a virtual right control 824.

Turning now to FIG. 9, a driving game specific virtual control interface900 is shown, in accordance with an aspect of the present invention. Thereference buttons 930, 932, 934, and 936 are located in the upper rightcorner of the virtual control interface 900. These buttons can be usedinfrequently in some driving games and are placed in the top rightcorner to reserve more accessible screen space for more frequently usedcontrols. The reference buttons correlate directly to buttons availableon a gamepad.

The virtual control interface 900 may include controls that do notdirectly correlate to controls on a gamepad, or at least, the virtualgame controls may include a label that does not match control labels ona gamepad. For example, virtual downshift control 940, virtual speedcontrol 942, and virtual shift-up control 944 are not found on agamepad. Instead, a trigger may be used to control speed, and otherbuttons, such as a bumper, may shift gears. Similarly, the virtualthrottle control 946 and virtual break control 948, may correlatefunctionally with buttons on a gamepad but have specific labels thatmatch the present game. The virtual control interface 900 also includesa virtual form button 950.

On the upper left side of virtual control interface 900, virtual arrows910, 912, 914, and 916 can perform the functions associated with adirection pad on a gamepad. Game-specific controls comprise a virtuallook back control 920, a virtual steering control 922, and a virtualhandbrake control 924. Additional game-specific controls include avirtual camera switch control 926 and a replay control 928. Functionsassociated with these controls may be mapped to controls on a gamepad.

Turning now to FIG. 10, a fighting specific virtual control interface1000 is shown, in accordance with an aspect of the invention. Thevirtual control interface 1000 comprises a virtual stick control 1010 onthe lower left portion of the touch screen. In one aspect, the virtualstick control 1010 is dynamically located upon the user initiallytouching the left portion of the touch screen.

The virtual buttons on virtual control interface 1000 include virtualreference buttons, virtual bumpers, virtual triggers and virtual stickpushes. The virtual reference buttons include virtual reference button1020, virtual reference button 1022, virtual reference button 1024, andvirtual reference button 1026. The virtual stick pushes include virtualleft stick push 1028 and virtual right stick push 1038. The virtualtriggers include virtual left trigger 1030 and virtual right trigger1036. The virtual bumpers include virtual left bumper 1032 and virtualright bumper 1034.

In one aspect, interface 1000 is optimized for use on a flat surface,such as a table or lap. The user's left hand can manipulate the virtualstick control 1010 while the user's right hand can push the variousbuttons.

Turning now to FIG. 11, a single touch multifunction virtual control1100 is illustrated, in accordance with an aspect of the presentinvention. The virtual control 1100 can be part of a virtual controlinterface. The virtual control 1100 can be manipulated by a singlefinger 1114 or a thumb and can perform multiple functions. The virtualcontrol 1100 can serve as a virtual stick control that manipulates acharacter's direction of movement or camera orientation. Directionalinstructions can be provided by moving the head 1112 in a differentdirection within the control's base zone 1110. By moving the virtualhead 1112 outside of the control base zone 1110, a second function, suchas a bumper button, can be activated causing the character to run incertain games. In another aspect, a magnitude of speed is assignedaccording to a distance moved from the base zone 1110. This can simulatea trigger function on a gamepad, which has an order of magnitudedetermined based on an amount the trigger is depressed.

Turning now to FIG. 12, an additional function implemented by the singletouch multifunction virtual control 1100 is illustrated, in accordancewith an aspect of the present invention. As previously illustrated inFIG. 11, moving a distance away from the base may assign a magnitude orspeed to the direction control. The magnitude zone 1210 is shown in FIG.12. Dragging the head 1212 outside of the magnitude zone 1210 mayactivate a second function, such as firing or jumping. In one aspect,moving the head 1212 outside the magnitude zone 1210 can still affectmovement of the character or camera view.

Turning now to FIG. 13, a single touch virtual control 1300 isillustrated, in accordance with an aspect of the present invention. Thevirtual control 1300 can be part of a virtual control interface. Thesingle touch virtual control 1300 is designed for manipulation by asingle finger 1330 or thumb. The single touch virtual control 1300comprises a virtual stick control 1320. Maintaining contact with thevirtual stick control 1320 causes a character to move or a camera viewto change. Tapping the finger 1330 anywhere outside of the virtual stickcontrol 1320 within the right zone 1310 activates an additionalfunction, such as firing. Tapping in the left zone 1315 could activate adifferent function. Though not shown, the left zone 1315 could have itsown multi-function single touch virtual control.

Turning now to FIG. 14, customization of a virtual control 1300 isillustrated. The virtual stick control 1320 can be moved within theright zone 1310 by dragging 1425 from its previous location. In oneaspect, the virtual stick control 1320 is tapped first to activate arelocation mode and then dragged to his desired location. As analternative, the virtual stick control 1320 could be located at thefirst location touched by the user within the right zone 1310.

Turning now to FIG. 15, a virtual WASD control interface 1500 is shown.A WASD interface emulates a control scheme that originates on a keyboardwhere the “A” button is associated with a left movement, the “W” key isassociated with up movement, the “D” is associated with a rightmovement, and the “S” is associated with a down movement. The virtualWASD interface 1500 can be combined with other controls, for example,controls manipulated by a thumb.

The middle finger 1512 can activate the up function 1522 or the downfunction 1524 by sliding back and forth or tapping on a function. Thevirtual control 1500 can be centered on a touch by the middle finger.Interactions by the index finger 1510 can activate function 1520.Interactions by the ring finger 1514 can activate function 1526.

Turning now to FIG. 16, a method 1600 for using a touch screen computingdevice to control a video game is provided, in accordance with an aspectof the present invention. Method 1600 may be performed by computer codebeing executed by one or more processors within a touch screen device,such as a smartphone or tablet.

At step 1610, optionally, a video game is determined to be active withina game execution environment on a gaming device, on the touch screendevice, or in a remote gaming service. In one aspect, the determinationmay be made by analyzing state information describing the gaming device.A gaming device can periodically provide state information to the touchscreen device. The virtual control interface application running on thetouch screen device may communicate with the gaming device periodically.A remote gaming service, such as gaming service 340 may facilitate thecommunication of state information. The state information can includeinformation that indicates the videogame is active.

At step 1620, at the touch screen computing device, a virtual controlinterface that is optimized for use with the video game is selectedbased on a characteristic of the video game. The virtual controlinterface application may select a suitable virtual control interfacefor a library of virtual control interfaces. The library may be a remotelibrary located at an online gaming service or on the gaming device. Thelibrary can be stored locally on the touch screen device. Either way, asuitable virtual control interface is selected based on a characteristicof the game, such as the game genre or game title. As describedpreviously, virtual control interfaces may be customized for aparticular game title or game genre, such as racing, fighting, strategy,first-person shooter, third-person shooter, and the like.

When making a selection, the virtual control interface application mayalso take into account user's preferences. The user's preferences may beexplicitly established by a user that associates a game title or gamegenre with a desired virtual control interface. The user's preferencescan be implicitly derived from the user's previous choices. For example,if the user played the game title with a first virtual controlinterface, then the first virtual control interface may be selected thenext time the user plays the game title. Similarly, if the user played aracing game with the first virtual control interface, then the firstvirtual control interface could be selected the next time the user playsa racing game, even if it is a different racing game.

In one aspect, the user may be presented with several virtual controlinterface options to select from. In this aspect, the selection can bethe result of direct user input. The user may be presented with ratings,comments, descriptions, and graphics that describe the various controlinterfaces the user can select from. In one aspect, a user's friend'spreferences or usage history is noted in the selection interface. Asmentioned, the various virtual control interfaces may be ranked in termsof relevance or popularity.

At step 1630, the virtual control interface is output for display on thetouch screen device's touch screen. The virtual control interfacecomprises a plurality of virtual game controls. The virtual controlinterface and virtual game controls have been described previously. Inone aspect, the touch screen device is not executing the videogame code.Instead, the videogame code is executed by a separate gaming device thatreceives control signals generated by the touch screen device. Forexample, the gaming device could be a game console or a PC.

In addition to virtual game controls, the virtual control interface maycomprise one or more annotations that help a user understand how thevirtual controls work. For example, an invitation could explained thatthe user “presses” a button by flicking their finger across the buttonperimeter. The annotation may take the form of words or graphics. Theannotation may be game title specific and provide a hint for overcominga particular challenge.

In one aspect, virtual game controls are dynamically located based onone or more user interactions. The virtual game controls may bedynamically moved during game play as the maximum range of the user'sfingers is ascertained. Detection of a smaller range may cause the sizeof the virtual game controls to be reduced and for them to be clumpedcloser together within the range. A larger range may cause the size ofthe virtual game controls to be expanded and for them to be distributeda greater distance from each other throughout the larger range.

In one aspect, some of the virtual game controls are fixed on aparticular location on the touch screen. Other virtual game controls maybe set when the user first interacts with the virtual control interface.For example, the virtual stick control can be located upon detectingwhen the user's thumb interacts with the virtual control interface.

At step 1640, an interaction with a virtual game control within thevirtual control interface is received through the touch screen. In oneaspect, the virtual game control corresponds to a gamepad control. Forexample, the virtual game control could be a virtual right triggercontrol that corresponds to the right trigger control on a gamepad. Asingle virtual control could also correspond to multiple gamepadcontrols. For example, a single-touch multifunction virtual control, asdescribed with reference to FIGS. 11 and 12, could correspond to both acontrol stick and a trigger on a gamepad. In another example, a singlevirtual control could correspond to activating multiple gamepad controlssimultaneously or in sequence.

At step 1650, in response to the interaction, a control signal isgenerated that comprises a functional instruction that is usable by thegame execution environment in which the game is running to manipulatethe video game. The control signal can take a similar form to a controlsignal generated by a gamepad and can require little to no translationby a receiving game execution environment. In another aspect, thecontrol signal takes a different form and is communicated over differentcommunication channel from a control signal generated by a gamepad. Forexample, the gamepad could generate infrared signal and the touch screendevice could generate a Bluetooth or Wi-Fi signal. The Bluetoothconnection could be directly between the touch screen device and thegaming device. The Wi-Fi signal may be communicated over a local networkor even across the Internet to a gaming service that routes the controlback to the gaming device.

At step 1660, the control signal is output to the gaming executionenvironment. For example, the control signal could be output from avirtual interface component on the touch screen device to the gameexecution environment on the touch screen device or the game executionon a separate gaming device. As mentioned, the control signal could beoutput through infrared, Bluetooth, Wi-Fi, or some other communicationsprotocol. The control signal may have destination informationidentifying the gaming device. The control signal could also identifythe touch screen device.

Turning now to FIG. 17, a method 1700 for using a touch screen computingdevice to control a video game is provided, in accordance with an aspectof the present invention. Method 1700 may be performed by computer codebeing executed by one or more processors within a touch screen device,such as a smart phone or tablet.

At step 1710, a virtual control interface, which comprises a pluralityof virtual game controls adapted to control the video game is output fordisplay on a touch screen device's touch screen. Examples of virtualcontrol interfaces and virtual game controls have been describedpreviously. In one aspect, the touch screen device is not executing thevideogame code. Instead, the videogame code is executed by a separategaming device that receives control signals generated by the touchscreen device. For example, a gaming device could be a game console or aPC. In another aspect, the touch screen device is both executing thevideogame code and providing the virtual control interface.

At step 1720, an interaction is received through the touch screen on avirtual game control within the virtual control interface. The virtualgame control corresponds to a gamepad control. For example, the virtualgame control could be a virtual right trigger control that correspondsto the right trigger control on a gamepad. A single virtual controlcould also correspond to multiple gamepad controls. For example, asingle-touch multifunction virtual control, as described with referenceto FIGS. 11 and 12, could correspond to both a control stick and atrigger on a gamepad. In another example, a single virtual control couldcorrespond to activating multiple gamepad controls simultaneous or insequence.

The interaction could be a single touch with an implement, such as afinger, thumb, or stylus. The interaction could be multitouch, whichcomprises contacting the touch screen in multiple places simultaneously.An interaction could also comprise a hover above the touch screen. Sometouch screens are able to detect the presence of a finger or otherimplement that are within a threshold distance of the touch screen. Inthe case of a hover, the point on the virtual control interfaceperpendicularly below the implement can be considered interacted with.

At step 1730, in response to the interaction, a control signal isgenerated that comprises a functional instruction that is able to beinterpreted by the video game as equivalent to an instruction receivedfrom a gamepad when the gamepad control is activated. The control signalcan take a similar form to a control signal generated by a gamepad andrequire little to no translation by a receiving game device. In anotheraspect, the control signal takes a different form and is communicatedover a communication channel different from a control signal generatedby a gamepad. For example, the gamepad could generate infrared signaland the touch screen device could generate a Bluetooth or Wi-Fi signal.The Bluetooth connection could be directly between the touch screendevice and the gaming device. The Wi-Fi signal may be communicated overa local network or even across the Internet to a gaming service thatroutes the control back to the gaming device.

At step 1740, the control signal is output to the video game. Asmentioned, the control signal could be output through infrared,Bluetooth, Wi-Fi, or some other communications protocol. The controlsignal may have destination information identifying the gaming device.The control signal could also identify the touch screen device.

Turning now to FIG. 18, a method 1800 for using a touch screen computingdevice to control a video game is provided, in accordance with an aspectof the present invention. Method 1800 may be performed by computer codebeing executed by one or more processors within a touch screen device,such as a smartphone or tablet.

At step 1810, a virtual control interface, which comprises a pluralityof virtual game controls adapted to control a video game running on aseparate gaming device, is output for display on a touch screen device'stouch screen. A virtual control interface and a virtual game controlhave been described previously. The virtual control interface controls avideo game running on a separate gaming device, such as a game console.

At step 1820, an interaction on a virtual game control within thevirtual control interface is received through the touch screen. Examplesof interactions have been described previously and can include touchingthe screen or hovering near the screen.

At step 1830, in response to the interaction, a visual indicia thatcommunicates the virtual game control was manipulated is output fordisplay on the touch screen. The visual indicia attempt to overcome thelack of tactile feedback provided by a gamepad. The visual indiciacommunicate that the virtual game control has been successfullyactivated. For example, upon pushing a green reference button, a greenglow may briefly appear around the perimeter of the screen or in someother portion of the screen.

At step 1840, in response to the touch input, a control signal isgenerated that comprises a functional instruction that is usable by thegaming device to manipulate the video game. The control signal can takea similar form to a control signal generated by a gamepad and requirelittle to no translation by a receiving game device. In another aspect,the control signal that takes a different form is communicated over adifferent communication channel from a control signal generated by agamepad. For example, the gamepad could generate infrared signal and thetouch screen device could generate a Bluetooth or Wi-Fi signal. TheBluetooth connection could be directly between the touch screen deviceand the gaming device. The Wi-Fi signal may be communicated over a localnetwork or even across the Internet to a gaming service that routes thecontrol back to the gaming device.

At step 1850, the control signal is wirelessly output. As mentioned, thecontrol signal could be output through infrared, Bluetooth, Wi-Fi, orsome other communications protocol. The control signal may havedestination information identifying the gaming device. The controlsignal could also identify the touch screen device.

In one aspect, the touch screen device may generate additional gameinput apart from interactions with the touch screen. For example,buttons on the touch screen device may be associated with a gamefunction or gamepad control. In one aspect, buttons located on the back,or side opposite of the touch screen, can be used to generate functionalcontrol signals. In one aspect, a touch surface is provided on the backof the touch screen device for the purpose of generating game controlsignals.

For example, the touch screen device could be a keyboard connected tothe touch screen device and folded behind it. Sections of the keyboardcould be associated with different commands; for example, a portion onthe right side of the keyboard could be associated with a virtual righttrigger control or a virtual right bumper control. In one aspect, thevirtual control interface on the touch screen provides a thumb optimizedvirtual game control on the right side and assigns a zone on the reartouch surface for interaction by the index finger and middle finger.

Aspects of the invention have been described to be illustrative ratherthan restrictive. It will be understood that certain features andsubcombinations are of utility and may be employed without reference toother features and subcombinations. This is contemplated by and iswithin the scope of the claims.

The invention claimed is:
 1. A method for using a touch screen computingdevice to control a video game, the method comprising: at the touchscreen computing device, selecting, based on a characteristic of thevideo game, a virtual control interface that is optimized for use withthe video game; outputting for display on the touch screen device'stouch screen the virtual control interface, which comprises a pluralityof virtual game controls; receiving through the touch screen aninteraction on a virtual game control within the virtual controlinterface; generating a control signal in response to the interactionthat comprises a functional instruction that is usable by a gameexecution environment to manipulate the video game; and outputting thecontrol signal to the game execution environment.
 2. The method of claim1, wherein the characteristic of the video game is the video game'sgenre.
 3. The method of claim 1, wherein the characteristic of the videogame is the video game's title.
 4. The method of claim 1, wherein thevirtual game control is displayed with an annotation describing how tointeract with the virtual game control.
 5. The method of claim 1,wherein the method further comprises receiving an initial interaction ata location on the virtual control interface and dynamically locating thevirtual game control at the location.
 6. The method of claim 1, whereinthe virtual game control is a fire control, and wherein the interactioncomprises an implement moving a distance from a first point within thefire control to a second point on the touch screen, and holding theimplement at the second point for a duration of time.
 7. The method ofclaim 6, wherein the distance controls a rate of discharge through aweapon with a longer distance increasing the rate of discharge.
 8. Acomputer system comprising: a touch screen display; a processor; and acomputer storage having computer-executable instructions embodiedthereon that when executed by the processor configure the computersystem to: output for display on the touch screen a virtual controlinterface, which comprises a plurality of virtual game controls adaptedto control the video game running on a separate gaming device; receivethrough the touch screen an interaction on a virtual game control withinthe virtual control interface; in response to the interaction, outputfor display on the touch screen a visual indicia that communicates thevirtual game control was manipulated; generate a control signal inresponse to the interaction that comprises a functional instruction thatis usable by the gaming device to manipulate the video game; and outputthe control signal wirelessly.
 9. The system of claim 8, wherein thevisual indicia is changing a portion of the touch screen to match acolor of the virtual game control.
 10. The system of claim 9, whereinthe portion of the touch screen is a perimeter of the touch screen. 11.The system of claim 8, further configured to: receive a touch through atouch interface located on a side opposite to the touch screen; generatea different control signal that comprises a different functioninstruction that is usable by the gaming device to manipulate the videogame; and output the different control signal wirelessly.
 12. The systemof claim 11, wherein the touch interface is a keyboard.
 13. The systemof claim 8, wherein the virtual game control corresponds to two or moregamepad controls.
 14. The system of claim 8, wherein the virtual gamecontrol is displayed with an annotation describing how to interact withthe virtual game control.
 15. One or more computer-storage media havingcomputer-executable instructions embodied thereon that when executed bya computing device perform a method for using a touch screen computingdevice to control a video game, the method comprising: outputting fordisplay on a touch screen device's touch screen a virtual controlinterface, which comprises a plurality of virtual game controls adaptedto control the video game running on a separate gaming device; receivingthrough the touch screen an interaction on a virtual game control withinthe virtual control interface; in response to the interaction,outputting for display on the touch screen a visual indicia thatcommunicates the virtual game control was manipulated; generating acontrol signal in response to the interaction that comprises afunctional instruction that is usable by the gaming device to manipulatethe video game; and outputting the control signal wirelessly.
 16. Themedia of claim 15, wherein the visual indicia is changing a portion ofthe touch screen to match a color of the virtual game control.
 17. Themedia of claim 16, wherein the portion of the touch screen is aperimeter of the touch screen.
 18. The media of claim 15, wherein themethod further comprises: receiving a touch through a touch interfacelocated on a side opposite to the touch screen; generating a differentcontrol signal that comprises a different function instruction that isusable by the gaming device to manipulate the video game; and outputtingthe different control signal wirelessly.
 19. The media of claim 18,wherein the touch interface is a keyboard.
 20. The media of claim 15,wherein the virtual game control corresponds to two or more gamepadcontrols.